Techniques for generating contextually-relevant recommendations

ABSTRACT

A recommendation engine generates contextually relevant media programs tailored to the specific context of the user. The recommendation engine interfaces with one or more Internet-of-Things (IoT) appliances to determine one or more food items stored within the IoT appliance(s) and/or one or more cooking capabilities associated with the IoT appliance(s). The recommendation engine also gathers profile data that reflects a set of preferences associated with the user. Based on the gathered data, the recommendation engine queries a database of cooking program data associated with various cooking programs. The recommendation engine obtains cooking program data that is contextually relevant to the user and then generates a contextual cooking program that describes how to perform a given recipe. During playback of the contextual cooking program, the recommendation engine can modify the contextual cooking program based on the progress of the user in following the recipe.

BACKGROUND Field of the Various Embodiments

The various embodiments relate generally to computer science andrecommendation engines and, more specifically, to techniques forgenerating contextually-relevant recommendations.

Description of the Related Art

A recommendation engine is a type of software program that recommendsitems to users. Recommendation engines are commonly implemented inconsumer-facing websites in order to assist users with finding itemsthat are relevant to those users. For example, a recommendation engineassociated with an online shoe retailer could recommend specific typesof shoes to different users. Among other things, recommendation enginescan reduce the extent to which users have to manually search forrelevant items and, therefore, can increase user engagement with varioustypes of consumer-facing websites, including product-oriented websites,service-oriented websites, and media-oriented websites, to name a few.

A recommendation engine associated with a consumer-facing website canimplement several techniques for recommending items to a user. Using onetechnique, the recommendation engine analyzes the browsing history ofthe user and then recommends items that are similar to the items theuser has previously viewed. For example, the recommendation engine coulddetermine that the user previously viewed different gardening tools andcould then recommend one or more additional gardening tools to the user.Using another technique, the recommendation engine analyzes the browsingbehavior of the user relative to one or more items and then recommendsitems based on user engagement with those items. For example, therecommendation engine could determine that the user typically does notpurchase items that cost more than one hundred dollars and thensubsequently recommend items that cost less than one hundred dollars tothe user. As a general matter, recommendation engines associated withconsumer-facing websites typically recommend items to users based onhistorical information associated with those users or similar users.

One problem with conventional recommendation engines is that, becausethe recommendations provided to users are generated based on historicalinformation, those recommendations can be out-of-date and not reflectthe current state or context of the user. For example, a conventionalrecommendation engine could recommend that the user purchase a certainitem, even though the user recently purchased that item and therefore nolonger needs the item. Alternatively, a conventional recommendationengine could recommend that the user watch a particular cooking showwhere the user is instructed how to prepare a certain dish, even thoughthe user does not have the necessary ingredients, utensils, appliances,and/or cooking skills to prepare that dish. Recommendations that areirrelevant to the user do not engage the user and, consequently, canreduce user engagement with consumer-facing websites whererecommendation engines are used.

Another problem with conventional recommendation engines is that,because the recommendations provided to users are generated based onhistorical information, those recommendations can become irrelevant whenthe state or context of the user changes. For example, suppose aconventional recommendation engine recommends that the user watch aninstructional video demonstrating how a sequence of steps should beperformed. If the user cannot perform certain steps as quickly asdescribed in the instructional video, then the instructional video couldend up advancing to subsequent steps that are not yet relevant to theuser. These types of situations also can reduce user engagement withconsumer-facing websites where recommendation engines are used.

As the foregoing illustrates, what is needed in the art is a moreeffective technique for generating contextually-relevant recommendationsto users.

SUMMARY

Various embodiments include a computer-implemented method for generatinga media program, including determining a first state that is associatedwith a first appliance or at least one item stored within the firstappliance, determining a first portion of media content based on thefirst state, wherein the first portion of media content includes a firstinstruction associated with the first state, and generating a firstmedia program based on the first portion of media content, wherein thefirst media program provides instructions to a user for performing oneor more operations associated with the first state.

At least one technical advantage of the disclosed techniques relative tothe prior art is that the disclosed techniques enable contextuallyrelevant media to be automatically generated for users. In the exampleof an instructional cooking program, a program generated using thedisclosed techniques may be more engaging to users than cooking programsrecommended to users via conventional recommendation engines.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the variousembodiments can be understood in detail, a more particular descriptionof the inventive concepts, briefly summarized above, may be had byreference to various embodiments, some of which are illustrated in theappended drawings. It is to be noted, however, that the appendeddrawings illustrate only typical embodiments of the inventive conceptsand are therefore not to be considered limiting of scope in any way, andthat there are other equally effective embodiments.

FIG. 1 illustrates a system configured to implement one or more aspectsof the various embodiments;

FIG. 2 is a more detailed illustration of the recommendation engine andcontent database of FIG. 1, according to various embodiments;

FIG. 3 illustrates how the recommendation engine of FIG. 1 generatescooking program data, according to various embodiments;

FIG. 4 is a flow diagram of method steps for generating cooking programdata, according to various embodiments;

FIG. 5 illustrates how the recommendation engine of FIG. 1 generates acontextual cooking program, according to various embodiments; and

FIG. 6 is a flow diagram of method steps for generating a contextualcooking program, according to various embodiments.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth toprovide a more thorough understanding of the various embodiments.However, it will be apparent to one skilled in the art that theinventive concepts may be practiced without one or more of thesespecific details.

As noted above, recommendation engines can be implemented to improveusability of various types of consumer-facing websites, includingproduct-oriented websites, service-oriented websites, and media-orientedwebsites. One problem with conventional recommendation engines is thatbecause the recommendations provided to users are generated based onhistorical information, those recommendations are often out of date andtherefore do not reflect the current state or context of the user.Another problem with conventional recommendation engines is that becausethe recommendations provided to users are generated based on historicalinformation, those recommendations can become irrelevant when the stateor context of the user changes. Recommendations that are irrelevant tothe user do not engage the user with consumer-facing websites whererecommendation engines are used, and in some cases, can reduce userengagement with such websites.

To address these issues, various embodiments include a recommendationengine that generates contextually relevant recommendations of mediaprograms for a user and/or generates customized media programs tailoredto the specific context of the user. The recommendation engineinterfaces with one or more Internet-of-Things (IoT) appliances togather appliance data that indicates one or more food items storedwithin the IoT appliance(s) and/or one or more cooking capabilitiesassociated with the IoT appliance(s), among other information. Therecommendation engine also gathers profile data that reflects a set ofpreferences associated with the user, one or more food items recentlypurchased by the user, and/or dietary constraints associated with theuser, among other information. The recommendation engine processes theappliance data and the profile data to generate context data indicatingthe overall state and/or context of the user.

Based on the context data, the recommendation engine queries a databasethat stores cooking program data associated with various cookingprograms. The cooking program data associated with a given cookingprogram includes media content associated with the cooking program, suchas audio/visual data related to a person or virtual entity performing aprocedure for preparing one or more food items. The cooking program dataalso includes a recipe that indicates step-by-step instructionscorresponding to that procedure as well as data indicating one or moreingredients, utensils, and/or appliances needed to implement the recipe.In one embodiment, the recommendation engine generates cooking programdata by analyzing one or more pre-existing cooking programs.

In response to querying the database, the recommendation engine obtainscooking program data that is contextually relevant to the user. Suchcontextually-relevant cooking program data may correspond to recipes forwhich the user is currently in possession of the needed ingredients,utensils, and/or appliances. Contextually-relevant cooking program datamay also correspond to recipes in which the user has expressed aninterest in following or recipes for which the user has the needed skilllevel to successfully perform.

Based on the cooking program data, the recommendation engine generates acontextual cooking program that includes media content describing how toperform the associated recipe. During playback of the contextual cookingprogram, the recommendation engine monitors the progress of the user infollowing the recipe and can modify the contextual cooking program, orplayback thereof, in order to maintain synchrony with and/or adapt tothe progress of the user. The recommendation engine also updates theprofile data associated with the user based on the progress of the userin following the recipe, one or more food preparation skills learned infollowing the recipe, and other data generated while the user followsthe recipe set forth in the contextual cooking program.

In one embodiment, the user may initially interact with therecommendation engine to specify various criteria for consideration whenrecommending and/or generating a recipe or cooking program. Suchcriteria could include, for example, reducing waste, prioritizingingredients that are locally-sourced, in season, and/or close toexpiration, targeting a specific range of preparation times, or adheringto a specific meal plan and/or recommended calorie/nutritional intake,among others. Then, the recommendation engine may combine the abovecriteria with any additional metadata associated with the user, such asa skill level, a flavor profile, and so forth, and generate and/orrecommend a set of cooking programs that are suited for the user at acurrent point in time. In this manner, the recommendation engine cangenerate or recommend contextual cooking programs and correspondingrecipes based on a wide variety of relevant data.

At least one technical advantage of the disclosed techniques relative tothe prior art is that the disclosed techniques enable contextuallyrelevant cooking programs to be automatically generated for users.Accordingly, cooking programs generated using the disclosed techniquesmay be more engaging to users than cooking programs recommended to usersvia conventional recommendation engines. Another technical advantage ofthe disclosed techniques is that the disclosed techniques enable acooking program to be dynamically adapted to the progress of the user infollowing a recipe set forth in the cooking program. By adapting thecooking program to each particular user, user engagement is maintainedor enhanced throughout the course of the cooking program. Yet anothertechnical advantage of the disclosed techniques is that the disclosedtechniques enable a cooking program to be optimized to fit a predefinedset of criteria, such as reduction of waste, environmental impact,nutritional intake, dietary needs, allergy considerations, flavorprofile, preparation time, skill level, and/or cost, so as to better fitthe particular needs of a given user. These techniques are broadlyapplicable to any technically feasible type of media beyond cookingprograms, although cooking programs are described herein for exemplarypurposes. These technical advantages represent one or more technologicaladvancements over prior art approaches.

System Overview

FIG. 1 illustrates a system configured to implement one or more aspectsof the various embodiments. As shown, a system 100 includes a clientdevice 110, a server device 130, and one or more Internet-of-Things(IoT) appliances 140 coupled together via a network 150. Client device110 is also coupled to one or more user input/output (I/O) devices 160,which includes a display device 162 and various input devices 166.Display device 162 and input devices 166 can be separate (e.g. a monitorand a keyboard/mouse) or integrated with one another (e.g. a touchscreen tablet).

Client device 110 or server device 130 may be any technically feasibletype of computer system, including a desktop computer, a laptopcomputer, a mobile device, a virtualized instance of a computing device,a distributed and/or cloud-based computer system, and so forth. IoTappliance(s) 140 may include any technically feasible type of householdappliance that is configured to communicate with client device 110and/or server device 130 via network 150, including a smartrefrigerator, a smart microwave, a smart range (electric or gas stove),a smart oven, and so forth. Network 150 may be any technically feasibleset of interconnected communication links, including a local areanetwork (LAN), wide area network (WAN), the World Wide Web, theInternet, or any combination thereof. In one embodiment, client device110 and server device 130 may be coupled together via the Internet, andclient device 110 and IoT appliance(s) 140 may be coupled together via alocal network, such as a WiFi network or Bluetooth.

As further shown, client device 110 includes a processor 112, I/Odevices 114, and a memory 116, coupled together. Processor 112 includesany technically feasible set of hardware units configured to processdata and execute software applications. For example, processor 112 couldinclude one or more central processing units (CPUs). I/O devices 114include any technically feasible set of devices configured to performinput and/or output operations, including, for example, a universalserial bus (USB) port, a high-definition multimedia interface (HDMI)port, or a serial interface, among others. Memory 116 includes anytechnically feasible storage media configured to store data and softwareapplications, such as, for example, a hard disk, a random-access memory(RAM) module, and a read-only memory (ROM). Memory 116 includes arecommendation engine 120(0), profile data 122 associated with a user,and context data 124. Recommendation engine 120(0) is configured togenerate context data 124 based, at least in part, on profile data 122,as described in greater detail below. In so doing, recommendation engine120(0) interoperates with a recommendation engine 120(1) that resides onserver device 130.

Server device 130 includes a processor 132, I/O devices 134, and amemory 136, coupled together. Processor 132 includes any technicallyfeasible set of hardware units configured to process data and executesoftware applications, such as one or more CPUs. I/O devices 134 includeany technically feasible set of devices configured to perform inputand/or output operations, such as various types of data ports. Memory136 includes any technically feasible storage media configured to storedata and software applications, such as, for example, a hard disk, a RAMmodule, and a ROM. Memory 136 includes recommendation engine 120(1)mentioned above as well as a content database 138.

In operation, recommendation engine 120(0) within client device 110interfaces with IoT appliance(s) 140 in order to gather various types ofappliance data, including food data 142 and capability data 144. For agiven IoT appliance 140, food data 142 indicates the types of food itemsstored in the IoT appliance 140 or the types of food items that can beprepared using the IoT appliance 140. For example, if the IoT appliance140 is a smart refrigerator, then food data 142 could indicate thedifferent food items currently stored in the smart refrigerator, theamounts of food items currently available, the expiration dates and/ornutritional information associated with those food items, and so forth.Alternatively, if the IoT appliance 140 is a smart microwave, then fooddata 142 could indicate the types of food items that can becooked/microwaved using the smart microwave. As a general matter, anygiven IoT appliance 140 can acquire information such as that describedabove using any existing techniques that IoT appliances use to acquiresuch information.

Capability data 144 indicates, for a given IoT appliance 140, thespecific food preparation capabilities associated with that IoTappliance 140. For example, if the IoT appliance 140 is a smart stove,then capability data 144 could indicate that the smart stove is capableof frying, boiling, simmering, and other food-oriented operations thatcan be performed using a stove. Further, capability data 144 couldindicate a temperature range available for various food preparationactivities, a food preparation capacity of the IoT appliance 140,available utensils that can be used in conjunction with the IoTappliance 140, and any other technically feasible data that describesthe operational capabilities of the IoT appliance 140.

Recommendation engine 120(0) also interacts with the user in order togather and/or generate profile data 122. Profile data 122 indicatesvarious preferences associated with the user, including food itempreferences, food preparation preferences, appliance preferences,cooking program preferences, and so forth. Profile data 122 alsoindicates various constraints associated with the user, includingdietary restrictions, food allergies, restrictions on cooking skilllevel, and so forth. Profile data 122 can also indicate a viewinghistory associated with the user relative to a video streaming service,a set of behaviors the user performs relative to the video streamingservice, and so forth. Based on food data 142, capability data 144, andprofile data 122, recommendation engine 120(0) generates context data124 to represent the overarching context or state within which the userperforms food preparation and cooking operations within a kitchenenvironment. Context data 124 may thus reflect the state of one or moreIoT appliances 140 and/or any item stored therein.

Recommendation engine 120(0) then transmits context data 124 torecommendation engine 120(1).

Recommendation engine 120(1) queries content database 138 based oncontext data 124 to obtain data records associated with cooking programsthat may be relevant to the user given the current context of the user,including media content associated with the preparation and/or cookingof food. In one embodiment, recommendation engine 120(1) may querycontent database 138 using a weighted combination of characteristics setforth in context data 124. The media content returned by contentdatabase 138 generally corresponds to recipes that call for ingredientsthe user currently has in possession, recipes that involve one or moreIoT appliances 140, recipes that align with the preferences of the userwithout violating any constraints of the user, and any other recipesthat can be associated with or derived from context data 124. The mediacontent may include one or more media clips, one or more media programs,one or more recipes associated with those media clips and/or mediaprograms, as well as data describing various ingredients, utensils, andappliances on which those recipes depend. In one embodiment,recommendation engine 120(1) populates context database 138 based on ananalysis of pre-existing cooking programs, as described in greaterdetail below in conjunction with FIGS. 2-4. Based on any or all of theabove described data, recommendation engine 120(1) generates acontextual cooking program 164 and transmits contextual cooking program164 to recommendation engine 120(0).

Contextual cooking program 164 may include live action portions, virtualportions rendered for display, and/or interactive portions that involveuser input. Recommendation engine 120(0) outputs contextual cookingprogram 164 to the user via display device 162. During playback ofcontextual cooking program 164, recommendation engine 120(0) monitorsthe progress of the user in following the recipe set forth in contextualcooking program 164. Recommendation engine 120(0) can then modifycontextual cooking program 164, or playback thereof, in order tomaintain synchrony between food preparation steps set forth incontextual cooking program 164 and food preparation steps beingperformed by the user, as described in greater detail below inconjunction with FIG. 5.

With the above techniques, recommendation engines 120(0) and 120(1)interoperate to generate cooking programs for users that arecontextually relevant. Accordingly, those cooking programs may be moreengaging to users than cooking programs suggested via conventionalrecommendation engines, thereby leading to greater user satisfaction.

In various embodiments, recommendation engines 120(0) and 120(1)represent separate portions of a distributed software entity,collectively referred to herein as recommendation engine 120, as shownin FIG. 2, where the inventive operations described herein aredistributed and/or balanced across client device 110 and sever device130 in any technically feasible fashion. In these embodiments, thedifferent portions of recommendation engine 120 may communicate with oneanother for the purposes of load balancing and/or work distribution,among other tasks, according to any existing technique(s). In othervarious embodiments, however, any or all of the inventive operationsdescribed herein can be performed by either recommendation engine 120(0)on client device 110 or recommendation engine 120(1) on server device130. For example, in some embodiments, recommendation engine 120(1) cangenerate context data 124.

Software Overview

FIG. 2 is a more detailed illustration of the recommendation engine andcontent database of FIG. 1, according to various embodiments. As shown,recommendation engine 120 includes a context generator 200 and a contentgenerator 202. As also shown, content database 138 includes cookingprogram data 210.

Context generator 200 is a software module that is configured to processprofile data 122, food data 142, and capability data 144 to generatecontext data 124. As discussed above, context data 124 represents theoverarching context or state within which the user performs foodpreparation and cooking operations. In one embodiment, context data 124may include a set of tags that correspond to characteristics associatedwith the user and/or one or more IoT appliance(s) 140 or any itemsincluded therein, a vector of data values that quantify thosecharacteristics, and/or a set of weight values indicating the relativeimportance of each characteristic. A given set of tags may be generateddynamically by context generator 200 or assigned by another entity.Context generator 200 provides context data 124 to content generator202.

Content generator 202 is a software module that is configured to querycontent database 138 based on context data 124 to identify cookingprogram data 210 that may be relevant to the user. In embodiments wherecontext data 124 includes a set of tags corresponding to characteristicsof the user and/or IoT appliances 140, content generator 202 may querycontent database 138 to identify cooking program data 210 that isassociated with the set of tags. Such tags can be generated or assignedduring generation of cooking program data 210. Content generator 202 maythen rank different sets of cooking program data 210 based on the degreeto which each set of cooking program data 210 corresponds to the set oftags.

As is shown, cooking program data 210 includes media content 212, one ormore recipes 214, and one or more dependencies 216(0) through 216(N)(hereinafter “dependencies 216”). Media content 212 includes audiovisualdata corresponding to a person or virtual entity performing foodpreparation and/or cooking operations, while recipe 214 delineates thosefood preparation and/or cooking operations. Dependencies 216 indicatevarious ingredients, utensils, and/or appliances on which recipe 214depends. Dependencies 216 may also indicate various food preparationand/or cooking skills that are needed to successfully follow recipe 214as well as potential sources of information for developing those skills,such as video tutorials. Each dependency 216 may be associated withmetadata indicating a priority level associated with the dependency, analternative version of that dependency, or one or more sources via whichthe dependency can be obtained. For example, a dependency 216corresponding to a particular ingredient could include metadataindicating that the ingredient is optional or required, metadataindicating a different ingredient that can be substituted in the recipe,and/or a location where the ingredient can be obtained.

In one embodiment, content generator 202 identifies relevant cookingprogram data 210 based on context data 124 by comparing context data 124to dependencies 216 and then extracting cooking program data 210 basedon whether the corresponding dependencies 216 are met. For example,suppose context data 124 indicates a set of food items that the usercurrently has available. Content generator 202 would thus only retrieveprogram cooking data 210 associated with recipes that call for the useof those food items and do not call for the use of food items that theuser does not currently have available. In another example, supposecontext data 124 indicates that the user does not have access to amicrowave. Content generator 202 would thus only retrieve cookingprogram data 210 associated with recipes that do not involve the use ofa microwave.

In one embodiment, the metadata associated with a given dependency 216may be used by recommendation engine 120 in order to automatically meetor resolve the given dependency 216 on behalf of the user, therebyallowing the user to follow the associated recipe. For example,recommendation engine 120 could determine that the user lacks a specificingredient corresponding to a given dependency 216 associated with aparticular recipe. Recommendation engine 120 could then automaticallyorder that specific ingredient for delivery to the user directly or viainteraction with one or more IoT appliances 140 configured to orderitems. In this manner, recommendation engine 120 resolves the givendependency 216 and provides the user with the needed ingredients tofollow the recipe. In other embodiments, recommendation engine 120 mayadd the item to an electronic shopping list along with a link to wherethe item can be purchased for delivery.

Content generator 202 generates contextual cooking program 164 based onsome or all of cooking program data 210. In some instances, contentgenerator 202 simply obtains media content 212 and then streams thatmedia content to the user. In more complex configurations, contentgenerator 202 dynamically generates contextual cooking program 164 bysplicing media content 212 with graphics rendered based on recipe 214and/or dependencies 216. Content generator 202 outputs contextualcooking program 164 to the user via display device 162. During playback,content generator 202 can dynamically modify contextual cooking program164, or playback thereof, based on the progress of the user inperforming the various steps associated with the recipe or based oninteractions with the user, as described in greater detail below inconjunction with FIG. 5. Content generator 202 can also spliceadditional interstitial content into contextual cooking program 164during periods of time when the user is waiting. For example, while theuser waits for a pot of water to boil, content generator 202 couldinsert an animated short or an advertisement into contextual cookingprogram 164.

In one embodiment, context generator 200 may implement a hybrid approachto generating contextually-relevant cooking programs independently orvia interoperation with content generator 202. In particular, contextgenerator 200 may implement knowledge-based filtering relative to fooddata 142 and capability data 144, collaborative filtering relative toviewing behavior and/or demographic data set forth in profile data 122,and content-based filtering based on a viewing history and/or associatedcontent tags set forth in profile data 122. Context generator 200 thengenerates a vector of data values to quantify various characteristicsdetermined through the above filtering operations. Based on the vectorof data values, content generator 202 can then dynamically weight and/oraugment each data value and query content database 138 to generate a setof recommendations of cooking program data 210 and/or associated mediacontent 212.

Recommendation engine 120 can be configured to generate cooking programdata 210 based on one or more source cooking programs 220 and/ordifferent portions thereof, as described in greater detail below inconjunction with FIGS. 3-4.

Generating Cooking Program Data

FIG. 3 illustrates how the recommendation engine of FIG. 1 generatescooking program data, according to various embodiments. As shown,recommendation engine 120 includes a content analyzer 300 that isconfigured to analyze different portions of a source cooking program 220related to a recipe 214 in order to generate cooking program data210(0). In the example shown, source cooking program portion 220(0)depicts a person using a knife 310 to chop a head of broccoli 312 whilea pot of water 314 boils. Source cooking program portion 220(1) depictsthe person adding chopped broccoli to pot, as well as message 316indicating that the broccoli should boil for 15 minutes. Message 316could be, for example, a closed captioning message or other graphicdepicting text.

Content analyzer 300 is configured to implement machine learning,computer vision, speech recognition and/or natural language processing,neural networks, and any other technically feasible form of artificialintelligence in order to identify the various food preparation and/orcooking operations set forth in source cooking program 220 and thendelineate those operations in recipe 214 as a set of instructions. As isshown, recipe 214 indicates two instructions, “chop broccoli” and “boil15 minutes.” In addition, content analyzer 300 implements any of theabove techniques to identify various dependencies 216 associated withrecipe 214, including dependency 216(0) indicating that broccoli isneeded, dependency 216(1) indicating that a knife 216(1) is needed, anddependency 216(2) indicating that a pot is needed. Content analyzer 300may also identify other dependencies associated with items notexplicitly shown in FIG. 3, such as a stove or a water source, forexample. Content analyzer 300 also parses source cooking program 220 togenerate media content 212. For example, content analyzer 300 coulddivide source cooking program 220 into media clips that correspond tospecific steps of recipe 214. Subsequently, content generator 202 coulddynamically recombine those media clips in varying sequences to generatecustomized cooking programs that reflect the skill level of the user,the progress of the user, and so forth. The techniques described aboveare described in greater detail below in conjunction with FIG. 4.

FIG. 4 is a flow diagram of method steps for generating cooking programdata, according to various embodiments. Although the method steps aredescribed in conjunction with the systems of FIGS. 1-3, persons skilledin the art will understand that any system configured to perform themethod steps in any order falls within the scope of the presentembodiments.

As shown, a method 400 begins at step 402, where content analyzer 300within recommendation engine 120 determines one or more ingredients,utensils, appliances, or operations used in source cooking program 220.In various embodiments, one or more of the above referenced items may beomitted. For example, some recipes 214 may not explicitly call for anyappliances. Recommendation 120 may further be configured to weight therelative importance of each dependency and indicate to the user whichdependencies can be safely omitted. Content analyzer 300 can alsoidentify specific skills that are needed to perform the identifiedoperations, determine the amount of time needed to perform thoseoperations, and/or determine quantities of ingredients needed whenperforming the operations. As a general matter, content analyzer 300 canimplement any technically feasible approach to identifying featureswithin audiovisual data when performing step 402, including machinelearning, computer vision, natural language processing, and/or neuralnetworks. FIG. 3 sets forth an example of how content analyzer 300analyzes a source cooking program 220 in order to determine any of theabove-described data.

At step 404, content analyzer 300 generates one or more dependencies 216based on the ingredients, utensils, appliances, and operationsdetermined at step 402. A given dependency 216 generally relates tosomething that is needed to perform the various operations set forth insource cooking program 220, including a specific amount of a giveningredient, a utensil needed to perform those operations, and/or anappliance used in performing the operations. A given dependency 216 mayalso relate to a cooking skill that is needed, an amount of time that isneeded, and any other data associated with cooking operations.

At step 406, content analyzer 300 generates a recipe 214 based on theoperations determined at step 402. Recipe 214 indicates a set ofinstructions corresponding to the operations performed in source cookingprogram 220. Content analyzer 300 can generate the set of instructionsby parsing text directly from source cooking program 220 or usingartificial intelligence to identify specific operations performed insource cooking program 220 and then generating instructions thatdescribe those operations. In various embodiments, content analyzer 300may import a recipe that meets a set of criteria corresponding to thevarious data determined via steps 402 and 404 (i.e. without explicitlygenerating any recipes) and may then recommend that recipe at step 406.

At step 408, content analyzer 300 generates media content 212 based onsource cooking program 220 and recipe 214. In doing so, content analyzer300 parses different portions of source cooking program 220 thatcorrespond to each instruction set forth in recipe 214. Each portion ofsource cooking program 220 depicts a person or virtual entity performinga food preparation and/or cooking operation associated with a giveninstruction. In various embodiments, content analyzer may generate mediacontent to graphically depict instructions set forth in recipe 214.

At step 410, content analyzer 300 combines the dependencies 216, recipe214, and media content 212 generated at steps 404, 406, and 408,respectively, into cooking program data 210. Content analyzer 300 canthen populate content database 138 with data records that includecooking program data 210. Subsequently, content generator 202 can querycontent database 138 for cooking program data 210 that is relevant tothe user and dynamically generate contextual cooking program 164 basedon identified cooking program data 210, as described in greater detailbelow in conjunction with FIGS. 5-6.

Generating a Contextual Cooking Program

FIG. 5 illustrates how the recommendation engine of FIG. 1 generates acontextual cooking program, according to various embodiments. As shown,a user 500 resides within a kitchen area 510 that includes an IoT stove512 and an IoT refrigerator 514. IoT refrigerator 514 stores food item516, which is a head of broccoli in this example. In one embodiment,kitchen area 510 is a smart kitchen that includes various input andoutput devices in order to both monitor the actions of user 500 andoutput various information to user 500.

In exemplary operation, recommendation engine 120 interfaces with IoTstove 512 and IoT refrigerator 516 in order to determine the state ofthose IoT appliances, including the food preparation capabilities of IoTstove 512 and the specific food items stored within IoT refrigerator514. Recommendation engine 120 then queries content database 138 in themanner described previously in order to determine cooking program data210 that may be relevant to user 500. Such cooking program data couldinclude, for example, a recipe 214 for preparing broccoli using a stove.

Recommendation engine 120 then generates contextual cooking program 164based on media content 212 included in the identified cooking programdata 210. Recommendation engine 120 can divide contextual cookingprogram 164 into segments 164(0) through 164(3) that correspond todifferent operations that user 500 is supposed to perform when followingrecipe 214 set forth in contextual cooking program 164. Recommendationengine 120 can then incrementally output each segment based on theprogress of user 500 in following recipe 214. In doing so,recommendation engine 120 generates progress data 520 to indicate aspecific instruction associated with recipe 214 that user 500 iscurrently performing. Recommendation engine 120 can identify thespecific instruction the user is currently performing based oninteractions with IoT appliances 140, based on explicit user input,and/or using computer vision or other relevant techniques.Recommendation engine 120 then modifies playback of contextual cookingprogram 164 to maintain synchrony with the progress of user 500. In theexample shown, recommendation engine 120 determines that user 500 is onthe first step of recipe 214, and therefore outputs segment 164(0) ofcontextual cooking program 164 in order to instruct user 500 to obtainfood item 516. Recommendation engine 120 can implement a similartechnique in order to advance a playback position to a relevant segmentbased on the progress of user 500. In this manner, recommendation engine120 implements a feedback loop in order to dynamically modify contextualcooking program 164, or playback thereof. These techniques are describedin greater detail below in conjunction with FIG. 6.

FIG. 6 is a flow diagram of method steps for generating a contextualcooking program, according to various embodiments. Although the methodsteps are described in conjunction with the systems of FIGS. 1-2 and 5,persons skilled in the art will understand that any system configured toperform the method steps in any order falls within the scope of thepresent embodiments.

As shown, a method 600 begins at step 602, where context generator 200within recommendation engine 120 obtains profile data 122 from the user.Profile data 122 indicates various preferences associated with the user,including food item preferences, food preparation preferences, appliancepreferences, cooking program preferences, and so forth. Profile data 122also indicates various constraints associated with the user, includingdietary restrictions, food allergies, restrictions on cooking skills,and so forth.

At step 604, context generator 200 obtains food data 142 and/orcapability data 144 from IoT appliance(s) 140. For a given IoT appliance140, food data 142 indicates the types of food items stored in the IoTappliance 140 or the types of food items that can be prepared using theIoT appliance 140. Capability data 144 indicates, for a given IoTappliance 140, the specific food preparation capabilities associatedwith that IoT appliance 140. Any of the above data may reflect the statea given IoT appliance 140 and/or any food items stored therein.

At step 606, context generator 200 generates context data 124 based onprofile data 122, food data 142, and/or capability data 144. In oneembodiment, context data 124 may include a set of tags that correspondto characteristics associated with the user and/or one or more IoTappliance(s) 140, a vector of data values, where each data valueindicates a score for a different characteristic associated with theuser and/or one or more IoT devices 140, and a set of weight values foreach data value. Further, context generator 200 may be configured todynamically modify the vector of data values and/or corresponding weightvalues over time based on user behavior, including viewing behavior,food consumption, food preparation activities, and so forth.

At step 608, content generator 202 within recommendation engine 120queries content database 138 to identify cooking program data 210corresponding to context data 124. In one embodiment, content generator202 may identify relevant cooking program data 210 based on context data124 by comparing context data 124 to dependencies 216 included incooking program data 210 and then extracting different cooking programdata 210 based on whether the corresponding dependencies 216 are met.

At step 610, content generator 202 generates contextual cooking program164 based on cooking program data 210 and outputs contextual cookingprogram 164 to the user via display device 162. Content generator 202can perform various techniques for generating contextual cooking program164, including stitching together one or more media clips included inmedia content 212, compositing portions of media content 212 withrendered text or graphics derived from recipe 214 and/or dependencies216, or simply obtaining media content 212 and outputting media content212 directly to the user.

At step 612, content generator 202 modifies contextual cooking program164 based on the progress of the user in following recipe 214 set forthin contextual cooking program 164. In particular, content generator 202can dynamically stall or advance playback of contextual cooking program164 based on the progress of the user in performing the various stepsassociated with recipe 214. Content generator 202 can also spliceadditional interstitial content into contextual cooking program 164during periods of time when the user is waiting. In one embodiment,content generator 202 monitors the progress of the user using one ormore input devices and then determining the real-time state of the userusing artificial intelligence, including machine learning, computervision, and/or neural networks.

In sum, a recommendation engine generates contextually relevant mediaprograms tailored to the specific context of the user. Therecommendation engine interfaces with one or more Internet-of-Things(IoT) appliances to gather appliance data that indicates one or morefood items stored within the IoT appliance(s) and/or one or more cookingcapabilities associated with the IoT appliance(s), among otherinformation. The recommendation engine also gathers profile data thatreflects a set of preferences associated with the user. Based on thecontext data, the recommendation engine queries a database that storescooking program data associated with various cooking programs. Based onidentified cooking program data, the recommendation engine generates acontextual cooking program that includes media content describing how toperform the associated recipe. During playback of the contextual cookingprogram, the recommendation engine monitors the progress of the user infollowing the recipe and can modify the contextual cooking program, orplayback thereof, in order to maintain synchrony with and/or adapt tothe progress of the user.

At least one technical advantage of the disclosed techniques relative tothe prior art is that the disclosed techniques enable contextuallyrelevant cooking programs to be automatically generated for users.Accordingly, cooking programs generated using the disclosed techniquesmay be more engaging to users than cooking programs recommended to usersvia conventional recommendation engines. Another technical advantage ofthe disclosed techniques is that the disclosed techniques enable acooking program to be dynamically adapted to the progress of the user infollowing a recipe set forth in the cooking program. By adapting thecooking program to each particular user, user engagement is maintainedor enhanced throughout the course of the cooking program. Thesetechnical advantages represent one or more technological advancementsover prior art approaches.

1. Some embodiments include a computer-implemented method for generatinga media program, the method comprising determining a first state that isassociated with a first appliance or at least one item stored within thefirst appliance, determining a first portion of media content based onthe first state, wherein the first portion of media content includes afirst instruction associated with the first state, and generating afirst media program based on the first portion of media content, whereinthe first media program provides instructions to a user for performingone or more operations associated with the first state.

2. The computer-implemented method of clause 1, further comprisingdetermining that the user has performed a first operation correspondingto the first instruction, and in response, advancing the first mediaprogram to a playback position corresponding to a second instruction.

3. The computer-implemented method of any of clauses 1-2, furthercomprising determining that the user is performing a first operationcorresponding to the first instruction, determining a first interval oftime corresponding to the first operation and during which the userperforms the first operation, pausing playback of the first mediaprogram during the first interval of time, and playing at least aportion of interstitial content during the first interval of time.

4. The computer-implemented method of any of clauses 1-3, furthercomprising determining a first dependency associated with the firstinstruction based on a source media program that includes the firstportion of media content, wherein the first dependency indicates a skillor item that is needed to implement the first instruction, generating afirst data record that includes the first portion of media content, thefirst instruction, and the first dependency, and populating a databasewith the first data record.

5. The computer-implemented method of any of clauses 1-4, whereindetermining the first portion of media content comprises executing aquery against the database using the first state to identify the firstdata record.

6. The computer-implemented method of any of clauses 1-5, wherein thefirst appliance comprises a food storage appliance, the at least oneitem comprises at least one food item, and the first appliance storesthe at least one food item.

7. The computer-implemented method of any of clauses 1-6, wherein thefirst state indicates an expiration date associated with the at leastone food item.

8. The computer-implemented method of any of clauses 1-7, wherein thefirst appliance comprises a food preparation appliance, and the firststate comprises a food preparation capability associated with the firstappliance.

9. The computer-implemented method of any of clauses 1-8, wherein thefirst state indicates a temperature range associated with the foodpreparation capability, wherein the first appliance performs foodpreparation operations associated with the temperature range.

10. The computer-implemented method of any of clauses 1-9, wherein thefirst appliance comprises a kitchen appliance, and the first mediaprogram comprises a cooking program.

11. Some embodiments include a non-transitory computer-readable mediumstoring program instructions that, when executed by a processor, causethe process to generate a media program by performing the steps ofdetermining a first state that is associated with a first appliance orat least one item stored within the first appliance, determining a firstportion of media content based on the first state, wherein the firstportion of media content includes a first instruction associated withthe first state, and generating a first media program based on the firstportion of media content, wherein the first media program providesinstructions to a user for performing one or more operations associatedwith the first state.

12. The non-transitory computer-readable medium of clause 11, furthercomprising the steps of determining that the user has performed a firstoperation corresponding to the first instruction, and in response,advancing the first media program to a playback position correspondingto a second instruction.

13. The non-transitory computer-readable medium of any of clauses 11-12,further comprising the steps of determining that the user is performinga first operation corresponding to the first instruction, determining afirst interval of time corresponding to the first operation and duringwhich the user performs the first operation, pausing playback of thefirst media program during the first interval of time, and playing atleast a portion of interstitial content during the first interval oftime.

14. The non-transitory computer-readable medium of any of clauses 11-13,further comprising the steps of determining a first dependencyassociated with the first instruction based on a source media programthat includes the first portion of media content, wherein the firstdependency indicates a skill or item that is needed to implement thefirst instruction, generating a first data record that includes thefirst portion of media content, the first instruction, and the firstdependency, and populating a database with the first data record.

15. The non-transitory computer-readable medium of any of clauses 11-14,wherein the step of determining the first portion of media contentcomprises executing a query against the database using the first stateto identify the first data record.

16. The non-transitory computer-readable medium of any of clauses 11-15,wherein the first instruction is associated with performing at least onefood preparation operation for a recipe being presented to a user viathe first media program.

17. The non-transitory computer-readable medium of any of clauses 11-16,further comprising the steps of determining a first dependencyassociated with the first instruction that indicates a skill or itemthat is needed to implement the first instruction, performing a firstaction to provide the user with the skill or item.

18. The non-transitory computer-readable medium of any of clauses 11-17,wherein the step of performing the first action comprises displaying asecond portion of media content to the user, wherein the second portionof media content indicates one or more techniques for performing theskill.

19. The non-transitory computer-readable medium of any of clauses 11-18,wherein the step of performing the first action comprises automaticallyobtaining the item from a third-party for the user.

20. Some embodiments include a system, comprising a memory storing asoftware application, and a processor that, when executing the softwareapplication, is configured to generate a media program by performing thesteps of determining a first state that is associated with a firstappliance or at least one item stored within the first appliance,determining a first portion of media content based on the first state,wherein the first portion of media content includes a first instructionassociated with the first state, and generating a first media programbased on the first portion of media content, wherein the first mediaprogram provides instructions to a user for performing one or moreoperations associated with the first state.

Any and all combinations of any of the claim elements recited in any ofthe claims and/or any elements described in this application, in anyfashion, fall within the contemplated scope of the present embodimentsand protection.

The descriptions of the various embodiments have been presented forpurposes of illustration, but are not intended to be exhaustive orlimited to the embodiments disclosed. Many modifications and variationswill be apparent to those of ordinary skill in the art without departingfrom the scope and spirit of the described embodiments.

Aspects of the present embodiments may be embodied as a system, methodor computer program product. Accordingly, aspects of the presentdisclosure may take the form of an entirely hardware embodiment, anentirely software embodiment (including firmware, resident software,micro-code, etc.) or an embodiment combining software and hardwareaspects that may all generally be referred to herein as a “module,” a“system,” or a “computer.” Furthermore, aspects of the presentdisclosure may take the form of a computer program product embodied inone or more computer readable medium(s) having computer readable programcode embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

Aspects of the present disclosure are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine. The instructions, when executed via the processor ofthe computer or other programmable data processing apparatus, enable theimplementation of the functions/acts specified in the flowchart and/orblock diagram block or blocks. Such processors may be, withoutlimitation, general purpose processors, special-purpose processors,application-specific processors, or field-programmable gate arrays.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

While the preceding is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

1. A computer-implemented method for generating a media program, themethod comprising: determining a first state that is associated with aprocedure to be performed by a user, wherein the first state comprises acapability of a first appliance or an availability of at least one itemstored within the first appliance; determining a set of instructions forperforming the procedure based on a comparison of the first state with aset of dependencies for the procedure; determining a first portion ofmedia content that depicts a first instruction included in the set ofinstructions; generating a first media program based on the firstportion of media content, wherein the first media program provides thefirst instruction to the user.
 2. The computer-implemented method ofclaim 1, further comprising: determining that the user has performed afirst operation corresponding to the first instruction; and in response,advancing the first media program to a playback position correspondingto a second instruction included in the set of instructions.
 3. Thecomputer-implemented method of claim 1, further comprising: determiningthat the user is performing a first operation corresponding to the firstinstruction; determining a first interval of time during which the userperforms the first operation; pausing playback of the first mediaprogram during the first interval of time; and playing at least aportion of interstitial content during the first interval of time. 4.The computer-implemented method of claim 1, further comprising:determining a first dependency associated with the first instructionbased on a source media program that includes the first portion of mediacontent, wherein the first dependency indicates a skill or item that isneeded to implement the first instruction; generating a first datarecord that includes the first portion of media content, the firstinstruction, and the first dependency; and populating a database withthe first data record.
 5. The computer-implemented method of claim 4,wherein determining the set of instructions comprises executing a queryagainst the database using the first state to identify the first datarecord.
 6. The computer-implemented method of claim 1, wherein the firstappliance comprises a food storage appliance, the at least one itemcomprises at least one food item, and the first appliance stores the atleast one food item.
 7. The computer-implemented method of claim 6,wherein the first state indicates an expiration date associated with theat least one food item.
 8. The computer-implemented method of claim 1,wherein the first appliance comprises a food preparation appliance, andthe first state comprises a food preparation capability associated withthe first appliance.
 9. The computer-implemented method of claim 8,wherein the first state indicates a temperature range associated withthe food preparation capability, wherein the first appliance performsfood preparation operations associated with the temperature range. 10.The computer-implemented method of claim 1, wherein the first appliancecomprises a kitchen appliance, and the first media program comprises acooking program.
 11. A non-transitory computer-readable medium storingprogram instructions that, when executed by a processor, cause theprocessor to generate a media program by performing the steps of:determining a first state that is associated with a procedure to beperformed by a user, wherein the first state comprises a capability of afirst appliance or an availability of at least one item stored withinthe first appliance; determining a set of instructions for performingthe procedure based on a comparison of the first state with a set ofdependencies for the procedure; determining a first portion of mediacontent that depicts a first instruction included in the set ofinstructions; generating a first media program based on the firstportion of media content, wherein the first media program provides thefirst instruction to the user
 12. The non-transitory computer-readablemedium of claim 11, wherein the program instructions further cause theprocessor to perform the steps of: determining that the user hasperformed a first operation corresponding to the first instruction; andin response, advancing the first media program to a playback positioncorresponding to a second instruction included in the set ofinstructions.
 13. The non-transitory computer-readable medium of claim11, wherein the program instructions further cause the processor toperform the steps of: determining that the user is performing a firstoperation corresponding to the first instruction; determining a firstinterval of time during which the user performs the first operation;pausing playback of the first media program during the first interval oftime; and playing at least a portion of interstitial content during thefirst interval of time.
 14. The non-transitory computer-readable mediumof claim 11, wherein the program instructions further cause theprocessor to perform the steps of: determining a first dependencyassociated with the first instruction based on a source media programthat includes the first portion of media content, wherein the firstdependency indicates a skill or item that is needed to implement thefirst instruction; generating a first data record that includes thefirst portion of media content, the first instruction, and the firstdependency; and populating a database with the first data record. 15.The non-transitory computer-readable medium of claim 14, wherein thestep of determining the set of instructions comprises executing a queryagainst the database using the first state to identify the first datarecord.
 16. The non-transitory computer-readable medium of claim 11,wherein the first instruction is associated with performing at least onefood preparation operation for a recipe being presented to a user viathe first media program.
 17. The non-transitory computer-readable mediumof claim 11, wherein the program instructions further cause theprocessor to perform comprising the steps of: determining a firstdependency associated with the first instruction that indicates a skillor item that is needed to implement the first instruction; andperforming a first action to provide the user with the skill or item.18. The non-transitory computer-readable medium of claim 17, wherein thestep of performing the first action comprises displaying a secondportion of media content to the user, wherein the second portion ofmedia content indicates one or more techniques for performing the skill.19. The non-transitory computer-readable medium of claim 17, wherein thestep of performing the first action comprises automatically ordering theitem for delivery to the user.
 20. A system, comprising: a memorystoring a software application; and a processor that, when executing thesoftware application, is configured to generate a media program byperforming the steps of: determining a first state that is associatedwith a procedure to be performed by a user, wherein the first statecomprises a capability of a first appliance or an availability of atleast one item stored within the first appliance; determining a set ofinstructions for performing the procedure based on a comparison of thefirst state with a set of dependencies for the procedure; determining afirst portion of media content that depicts a first instruction includedin the set of instructions; and generating a first media program basedon the first portion of media content, wherein the first media programprovides the first instruction to the user.